AdvertisementAdvertisementAI is undermining the web's grand bargain, and a decades-old handshake agreement is the only thing standing in the way. Now, though, generative AI and large language models are changing the mission of web crawlers radically and rapidly. Without a supply of potential consumers, there's little incentive for content creators to let web crawlers continue to suck up free data online. It's also open to manipulation, especially given the voracious appetite for quality AI data. Because robots.txt is voluntary, web crawlers can also simply ignore the blocking instructions and siphon the information from a site anyway.

---

Persons: Microsoft's Bing, Joost de Valk, It's, de Valk, Nick Vincent, Valk, OpenAI, robots.txt, Jason Schultz, Catherine Stihler, Archie, NYU's Schultz, Steven Sinofsky, who's, Andreessen Horowitz, De Valk, Stihler Organizations: Big Tech, Google, Wordpress, NYU's Technology, Policy Clinic, AWS, Creative Commons, Creative, Microsoft, Nvidia, Star Wars, DC Comics, Warner Brothers, Marvel, Disney, Atlantic, Meta Locations: CCBot, EleutherAI